Controlling Experiment-wise Type I Error of Meta-analysis in the Solomon Four-group Design

Stouffer=s Z, a meta-analytic technique, was proposed by W. Braver and Braver (1988) for analyzing data collected from a Solomon Four-group Design. Sawilowsky, Kelley, Blair, and Markman (1994) showed that this technique produces inflated Type I error rates. Recommendations are made to control the false positive inflation of their procedure. According to Campbell and Stanley (1963), the Solomon Four-group Design (Solomon, 1949) has “higher prestige” than most other quantitative research designs, because it permits “explicit consideration of external validity factors” (p. 24). It contains both pretested and nonpretested treatment and control groups. Campbell and Stanley noted the lack of a single statistic for analyzing all of the data collected from this design. To resolve this data analysis problem, W. Braver and Braver (1988) proposed the use metaanalysis. They provided a flowchart that documents a sequence of testing procedures. The first test (Test A) is a 2×2 ANOVA on the means of the four posttests= scores, which examines the interaction of Pretest Treatment. If this test is significant, pretest sensitization is indicated, which is an undesirable situation where the outcome is confounded by the effect of the pretest (see, e.g., Willson & Putnam, 1982; Lana, 1959). W. Braver and Braver (1988) suggested a variety of followup tests (called Test B and Test C) to further investigate the nature of the pretest effects. If Test A is not significant, the sequence continues with Test D, which is a test of main effects for the treatment vs. control group. Conditioned on these two preliminary tests being non-significant, the procedure continues with Test E, an ANCOVA on the posttest scores for the treatment and control group that was pretested, with the pretest scores used as the covariate. (W. Braver & Braver, 1988, suggested other tests, referred to as Test F and Test G, which could be substituted for the ANCOVA.) If the ANCOVA is not significant, the next step is to conduct an independent samples t test on the posttest scores for the non-pretested treatment and control groups, which they referred to as Test H. A significant result of Test E (ANCOVA) or Test H (t test) indicates a significant treatment outcome. If neither of the final two classical tests (Test E and Test H) are significant, the sequence of testing concludes with Test I, Stouffer=s Z (Stouffer, et al., 1949) which is a meta-analytic procedure. W. Braver and Braver (1988) noted that nothing precludes the use of meta-analysis in